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(57) Abstract 

A method of determining alertness of a subject including the steps of non-invasively monitoring the subject's eye dynamics; generating 
from monitored eye dynamics a first output which is a measure of the subject's gaze patterns; generating from monitored eye dynamics a 
second output which is a measure of the subject's saccadic movements; generating from monitored eye dynamics a third output which is 
a measure of the subject's pupil dynamics; generating from monitored eye dynamics a fourth output which is a measure of the subject's 
blink patterns, and using the first, second, third and fourth outputs to compute a measure of alertness of the subject 
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AN ALERTNESS MONITOR 
Background of the Invention 
The invention relates to a system and method for 
monitoring the alertness of a person, such as a driver of a 
5 vehicle. 

Diminished driver attention was found in various 
Department of Transportation (DOT) sponsored studies to be 
responsible for up to 50% of all accidents on US highways. 
The segment of these accidents most carefully studied is 

10 that involving heavy trucks. Total annual cost of all 

accidents involving trucks is estimated by the DOT as being 
nearly $9.0 billion. Inattention generally is implicated in 
30% to 50% of truck accidents (DOT figures) at a cost of 
$3.0 to 4.5 billion/year in the US alone. 

15 Diminished attention may be due to a variety of 

causes: emotional stress, boredom, environment, sleep 
history, food history, drugs, alcohol, and fatigue. 
However, the portion of truck accidents attributable to 
driver fatigue has received the most concentrated attention 

20 since this is the cause most easily isolated in police 

accident reports. Fatigue has been directly implicated in 
the various studies as accounting for between 5% and 10% of 
all truck accidents with an annual cost to the US 
approximately $0.5 to $1.0 billion. 

25 Since 1990 a significant number of studies and 

research programs have been completed by existing commercial 
and academic organizations. These programs have 
concentrated on identifying driver fatigue, whilst accepting 
that this represents only one part of the whole driver 

30 attention problem. Also, they have tended to use a variety 
of indirect measurements tied back to fatigue (e.g. steering 
wheel movement, lane position, and lateral acceleration), 
since no practical way of measuring diver alertness in the 
real world of a commercial truck cabin or an operators 

35 environment was known. 
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People have also developed comprehensive strategies 
for dealing with fatigue. Some of those strategies are 
based, for example, upon initially using the vehicle air 
conditioning and sound systems to alter the conditions of 
5 the environment. If these techniques fail to generate a 
response, a second step utilizes induced vibration of the 
operator's seat and steering wheel and possibly reduction in 
the engine throttle setting. 

There is, however, still a need for an alertness 
10 monitor that signals the deterioration of alertness soon 
enough to produce reliable corrective actions and that 
measures not merely fatigue but the broader and more 
important question of inattention. 

Summary of the Invention 

15 The invention is an alertness monitor which combines 

the measurement of blink patterns used in past programs with 
direct measurement of changes in saccadic movements (i.e., 
autonomous movement of the eye) , in pupil characteristics 
(e.g. diameter and hippus) , and in viewing strategies (e.g. 

20 the alert driver pans from windshield to mirrors to side 
view to instruments and back, the less alert the driver 
tends to tunnel vision of the road ahead) . 

In general, in one aspect, the invention is a method 
of determining alertness of a subject. The method includes 

25 the steps of non-invasively monitoring the subject's eye 
dynamics; generating from monitored eye dynamics a first 
output which is a measure of the subject's gaze patterns; 
generating from monitored eye dynamics a second output which 
is a measure the subject's saccadic movements; generating 

30 from monitored eye dynamics a third output which is a 

measure of the subject's pupil dynamics; generating from 
monitored eye dynamics a fourth output which is a measure of 
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the subject's blink patterns; and using the first, second, 
third and fourth outputs to compute a measure of alertness 
of the subject. 

In preferred embodiments, the method also includes 
5 the step of using the measure of alertness to generate an 
alarm. 

In general, in another aspect, the invention is an 
apparatus for determining alertness of a subject. The 
apparatus includes an eye tracker module which during 

10 operation non-invasively monitors the subjects' eye 
dynamics; and a processor module which analyzes the 
monitored eye dynamics and generates therefrom a measure of 
the subject's alertness. The processor module is programmed 
to implement the following functions: generate from 

15 monitored eye dynamics a first output which is a measure of 
the subject's gaze patterns; generate from monitored eye 
dynamics a second output which is a measure the subject's 
saccadic movements; generate from monitored eye dynamics a 
third output which is a measure of the subject's pupil 

20 dynamics; generate from monitored eye dynamics a fourth 

output which is a measure of the subject's blink patterns; 
and use the first, second, third and fourth outputs to 
compute said measure of alertness of the subject. 

In general, in yet another aspect, the invention is 

25 a method of determining alertness of a subject. The method 
includes the steps of non-invasively monitoring the 
subject's eye dynamics; generating from monitored eye 
dynamics three measures of the subject's eye dynamics, 
wherein the three measures are selected from the group 

30 consisting of the subject's gaze patterns, the subject's 
saccadic movements, the subject's pupil dynamics, and the 
subject's blink patterns; and using all three measures to 
compute a measure of alertness of the subject. 



WO 98/49028 PCT/US98/08185 



Other advantages and features will become apparent 
from the following description of the preferred embodiment 
and from the claims. 



Description of the Preferred Embodiments 
5 Fig. 1 is a block diagram of the alertness 

monitoring system. 

Structure and Operation 

Referring to Fig. 1, a system which embodies the 
invention includes a remotely mounted eye tracker 10, a 

10 computer or data processor 12 which analyzes the output of 
the eye tracker to generate a measure of alertness, and an 
alarm unit 13 which produces an alarm to notify the driver. 
Eye tracker 10, which includes a video camera 14 and an eye 
gaze monitor 16, is directed toward the location where the 

15 drivers head will be so that during operation it obtains an 
image of the driver's head and eyes 15. Typically, eye 
tracker 10, or at least the video camera component of it, is 
mounted on the dashboard in front of the driver and below 
the level of the driver's head. It is positioned so as to 

20 the view the driver's eyes and not be obstructed by any 
objects, such as the steering wheel or the driver's arms. 
The output of video camera 14 passes to eye gaze monitor 16, 
which continuously analyzes the video image of the driver's 
eye, extracts relevant information relating to eye dynamics, 

25 computes a measure of alertness from that information, and 
generates an alarm signal which causes alarm unit 13 to take 
corrective action, e.g. alert the driver, when the alertness 
measure indicates that the driver's alertness has dropped 
below a predetermined level. 

30 Eye gaze monitor 16 is programmed to extract from 

the image of the driver's eye measures relating to the 
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following four characteristics: gaze pattern, saccadic 
movements, pupil dynamics, and blinking characteristics. 
Measurable changes in each of these characteristics are 
linked to increased fatigue and diminished alertness and 
5 collectively they provide a good indicator of the driver's 
alertness. 

As the subject's alertness diminishes, his or her 
gaze strategies reflect this in the following ways. The 
area of the field covered by the subject's gaze pattern 

10 decreases - a phenomenon referred to as w gaze tunneling" . 
Also, the duration of the subject's fixations on objects or 
locations in the subject's visual field increases and the 
frequency of those fixations decreases. In addition, slow 
rolling eye movements begin to occur. 

15 Saccadic movements refer to the naturally occurring, 

involuntary, rapid, small movements of the eye which result 
in changing the point of fixation on a visualized object. 
As the subject's alertness decrease, the velocity profile of 
these rapid eye movements becomes skewed (i.e., the shape of 

20 the velocity profile changes) . Also, the average distance 
moved and the peak velocities which occur during these 
saccadic movements changes and the frequency of the saccades 
decreases, i.e., there are fewer fixations and saccades 
corresponding to less active scanning. 

25 Pupil dynamics reflect decreases in alertness by a 

reduction in the mean pupil diameter and by significant 
changes in the Hippus movements, which are the abnormally 
exaggerated, rhythmic contractions and dilations of the 
pupil which are independent of changes in illumination or in 

30 fixation of the eyes. 

Finally, the blink characteristics show the 
following changes in response to reduced alertness. The 
frequency of closure (i.e., blinks) increases during periods 
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of rapid blinking. The intervals between periods of rapid 
blinking increases. The 80% perclose values (i.e., the 
percentage of time that the eyelids are at least 80% closed) 
increases. The eyelid velocity profile during blinking 
5 changes. And the percentage or amount of pupil area 
obscured by the eyelids between blinks increases. 

Processor 12 executes an algorithm 17 which is 
stored in local computer readable memory. The algorithm 
combines the measures provided by monitor 16, preferably all 

10 four of them, to compute the measure of alertness. The 
algorithm can be implemented in several alternative ways. 
For example, one approach is to combine the four parameters 
in a regression equation to generate an output value that is 
the measure of alertness. In one embodiment, the regression 

15 equation would take the following form: 

Z = b 0 + EjbA 

where Z is the combined measure of alertness 

Xi for lsis4, are the four previously described 
measures of eye dynamics 

20 b 0 is an intercept coefficient; and 

bi is the coefficient of the i th independent variable 
X t ; and 

The coefficients b 0 and bi can be readily determined through 
empirical studies of alertness using another well recognized 

25 and acknowledged indicator of alertness, such as those 
produced by electroencephalography (EEG) , 
electrocardiography (ECG) , electromyography (EOM) , or 
electro-oculography (EOG) . In other words, the coefficients 
are computed so that the four measured variables when 

30 weighted by the coefficients produce an output that 
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corresponds to the output obtained from one of these other 
sources, preferably EEC 

Alternative approaches involve using the parameters 
sequentially. That is, a partial set of parameters is first 
5 used to produce a preliminary measure of alertness and the 
remaining parameters are then used to produce a more precise 
measure of alertness when the preliminary measure exceeds a 
first threshold. In that case, the eye monitor can also be 
instructed to produce only the measures that are needed. 

10 When the generated measures indicate that the full set of 
parameters are required, then the eye monitor can be 
instructed to provide the additional parameters. These 
approaches have the advantage of imposing a reduced 
computational load on the processor and on the eye monitor 

15 during periods when the driver is fully sufficiently alert. 
In the described embodiment, the alarm unit 
generates an audible alarm. Alternatively, it could notify 
the driver in any of a number of different ways, e.g. 
through light, sound, or vibration, to just name a few. 

20 Also, the alarm could be programmed to be progressively more 
intrusive if the driver does not react as desired, e.g. if 
the monitoring equipment does not . show an improvement in 
alertness- In addition, different alarms could be used 
depending on the success on the first or earlier alarms. 

25 For the eye tracker and components thereof, one can 

use commercially available off-the-shelf tracking equipment 
such as the Model ETS (Eye Tracker System) , the Series 5000 
eye trackers, and the Model 210 Eye Movement Monitor, all of 
which are commercially available from Applied Science 

3 0 Laboratories, Inc., the assignee of the present invention. 
Of course, the equipment which is currently available 
through commercial source is typically not optimally 
designed for use in automobiles or trucks, so it is 
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preferable to modify it by, for example, reducing its size 
and configuration so that it fits more conveniently and 
unobtrusively in the environment in which the alertness 
monitor is intended to be used. 
5 In the case of the Model ETS from Applied Science 

Laboratories as well as other commercially available eye 
trackers, the unit illuminates the eye with an infra-red 
beam, or near- infra red beam, and views the eye with a solid 
state video camera. Since the near-infra red (IR) light is 

10 outside of the visible spectrum, it does not annoy the 
subject (e.g. the driver) but is in a high sensitivity 
region for most solid state light sensors. The video 
information is sent to a computer or special purpose 
processor for feature recognition and computation tasks. 

15 Line of gaze is computed, independently of head 

translations, by detecting the relative position of two 
features, namely, the pupil and the reflection of the IR 
light source on the corneal surface. The latter feature is 
commonly called the corneal reflection, or CR. A known 

20 mapping and/or curve fitting technique is then used to 
transform the pupil -CR vector to a point of gaze on the 
scene space. Pupil diameter is generally a byproduct of 
this pupil recognition task. 

Some systems illuminate the eye with a near IR beam 

25 that is coaxial with the camera optical axis. In that case, 
the camera receives a retro reflection from the retina, and 
the pupil appears as a bright circle with respect to a 
generally darker background. Other systems use a light beam 
that is off axis from the camera yielding a pupil with the 

30 familiar appearance of a black circle. These systems look 
for a pupil that is generally darker than any similar shape 
in the video field. 
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In order to achieve sufficient image resolution and 
to simplify the pattern recognition task, many eye tracking 
systems use a telephoto image of the eye with a field of 
view having roughly a one inch diameter at the distance of 
5 the subject's face. The image area thus includes only the 
eye and a small portion of the surrounding eye lid 
structure. The resulting image is relatively uncomplicated, 
and features of interest cover a large number of pixels. 
Head motion is handled by moving the entire optics module, 

10 on a servo controlled platform, to keep the camera aimed at 
the eye or, alternatively, by bending the optical path with 
a servo controlled mirror to accomplish the same purpose. 

For driver alertness applications, it may be 
desirable, for reasons of dependability and cost, to use a 

15 wide angle camera and then to use known relatively simple 
pattern recognition processing to locate the subjects eye 
within the image field. 

Other embodiments are within the following claims. , 
For example, an effective altertness monitor can also be 

20 constructed using a subset of the four parameters described 
above. A workable subset includes gaze pattern, saccadic 
movements, and blinking characteristics. Thus, in an 
alternative embodiment, the alertness measure is derived 
from such a subset of measured eye dynamics. 

25 What is claimed is: 
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Claims : 

1. A method of determining alertness of a subject, 
said method comprising: 

non-invasively monitoring the subject's eye 
dynamics ; 

generating from monitored eye dynamics a first 
output which is a measure of the subject's gaze patterns; 

generating from monitored eye dynamics a second 
output which is a measure the subject's saccadic movements; 

generating from monitored eye dynamics a third 
output which is a measure of the subject's pupil dynamics; 

generating from monitored eye dynamics a fourth 
output which is a measure of the subject's blink patterns; 
and 

using the first, second, third and fourth outputs to 
compute a measure of alertness of the subject. 



2. The method of claim 1, further comprising using 
the measure of alertness to generate an alarm, 

3 . An apparatus for determining alertness of a 
subject, said apparatus comprising: 

an eye tracker module which during operation non- 
invasively monitors the subjects' eye dynamics; and 

a processor module which analyzes the monitored eye 
dynamics and generates therefrom a measure of the subject's 
alertness, said processor module programmed to implement the 
following functions: 

generate from monitored eye dynamics a first output 
which is a measure of the subject's gaze patterns; 

generate from monitored eye dynamics a second output 
which is a measure the subject's saccadic movements; 



WO 98/49028 



PCT/US98/08185 



- 11 - 

generate from monitored eye dynamics a third output 
which is a measure of the subject's pupil dynamics; 

generate from monitored eye dynamics a fourth output 
which is a measure of the subject's blink patterns; and 

use the first, second, third and fourth outputs to 
compute said measure of alertness of the subject. 

4. A method of determining alertness of a subject, 
said method comprising: 

non-invasively monitoring the subject's eye 
dynamics; 

generating from monitored eye dynamics three 
measures of the subject's eye dynamics, wherein the three 
measures are selected from the group consisting of the 
subject's gaze patterns, the subject's saccadic movements, 
the subject's pupil dynamics, and the subject's blink 
patterns; and 

using all three measures to compute a measure of 
alertness of the subject. 
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